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ABSTRACT 



The issue of evaluation in adult literacy is 



addressed in this paper. It is assumed that evaluation is a necessary 
and vital function in an adult literacy program because it helps 
administrators understand their efforts, increases information 
available about a program's functioning, builds an image, and 
facilitates the discovery of information on unexpected program 
results. Three major topics are the focus of the discussion: (1) the 
perceived barriers to conducting evaluations of adult literacy 
programs, such as measurement or design problems, staff issues, 
relevance, and resource issues; (2) the range of evaluations of 
literacy programs that have been undertaXen recently? and (3) 
recommendations for policies and practices that can be implemented to 
encourage the collection of useful and valid evaluation data, 
including disseminating information collected, requiring 
documentation of program results, encouraging volunteer literacy 
programs to collect data on program effectiveness, and clarifying 
incentives for conducting program evaluation. Contains 10 references. 
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Introduction 



Over the past decade the role of evaluation has become an increasingly 
more integral part of die delivery of educational programs. The emphasis on 
accountability from program flinders and the changin g priorities In program 
development have necessitated the documentation and verification of pro- 
gram results. This concern has permeated all areas of education and has re- 
ceived significant attention in the recent educational literature This paper ad- 
dresses the issue of evaluation in the area of adult literacy. Three major topics 
will be the focus of this discussion: (1) the perceived barriers to conducting 
evaluations of adult literacy programs; (2) the range of evaluations of literacy 
programs that have been undertaken most recently: and (3) policies and prac- 
tices that can be implemented to encourage the collection of useful and valid 
evaluation data. 

Assumptions 

As a point of departure for this discussion the assumption is made that 
evaluation is a necessary and vital function in an adult literacy program. Some 
reasons for this assumption are: 

• Evaluation can help administrators of adult literacy programs build an 
understanding of their efforts that might otherwise take years iden- 
tify through regular program activities; 

• Evaluation can rapidly increase the information that is available about a 
program's functioning and reduce the need for trial-and-error activity; 

• Evaluation can be used to build an image that ordinarily comes slowly 
through contact with the program by and through its participants; 

• Evaluation facilitates the discovery of information concerning unex- 
pected program results that otherwise might go unnoticed (Steele 1 977). 

Thus a key function of evaluation is to generate information that can be 
used as one of the bases for making decisions about program improvements. 
In addition to the program improvement area, evaluation results can be uti- 
* lized in the justification for the expenditure of program funds and in the argu- 
ment for additional program monies. As the movement toward creating joint 
ventures between education, business, and industry progresses, it will be nec- 
essary to collect data that speak to the utility and effectiveness of the joint pro- 
grams. All of these reasons support the assumption that evaluation plays a 
critical role in adult literacy programming, g 
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Definitions 

Ideas about evaluation have changed substantially over time with de- 
velopments in design and methodology. In the past evaluation was limited to 
determining whether content-specific objectives had been achieved. The 
creation of new types of programs has resulted in the conception of broader 
frameworks for determining program effectiveness. For the purposes of this 
paper, the following definition of evaluation will be utilized 

Program evaluation is the systematic accumulation of facts for providing in- 
formation about the achievement of program requisites and goals relative to 
efforts, effectiveness and efficiency within any stage of program develop- 
ment (Tripoli Feflin. and Epstein 1971. 1 12). 

A second concept that requires clarification for this discussion is adult lit- 
eracy. In this paper adult literacy refers to the skills and knowledge acquired 
in reading, writing, and life-coping areas. The population discussed is those 
adults who have not obtained a high school diploma. This perspective is taken 
in order to maximize the number of literacy programs that can be examined in 
terms of the evaluation strategies that they have utilized. 

Barriers to Evaluation 

When the question of how to determine the effectiveness of adult liter- 
acy programs is raised, what frequently follows is a discussion of the reasons 
why conducting such an evaluation is difficult or impossible. Some of these 
barriers are concrete, while others are perceptions that are net necessarily sub- 
stantiated by facts. Some of the more frequently died barriers are discussed at 
length. 1 

Measurement/Design Problems 

The key issue here is what to measure. It is difficult to construct a mean- 
ingful evaluation design once an adult literacy program has been operating for 
a period of time-. Since' the emphasis on measuring effectiveness is relatively 
new in adult literacy, in the 1970s many of the innovative programs were de- 
veloped with little attention given to the evaluation design What resulted 
was the creation of ex post facto designs for these programs, an effort that has 
involved a great deal of time and energy on the part of program staff and exter- 
nal consultants. Literacy programs that have been more evolutionary and re- 
sponsive to rbanging clientele have experienced difficulty in deciding what to 
focus on in conducting an assessment of outcomes. This is compounded by 
the fact that most literacy programs have multiple outcomes for their clients. 
The issue of how to measure is complex and is one of the areas where percep- 
tions may be a more significant barrier than the actual methodology. The gen- 
eral perception among adult educators is that a traditional experimental 
design is the only acceptable methodology to use in program evaluations. 
Given the structure of adult literacy programs with open entry and exit, and 
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the ethical issues involved in withholding services from a client, it is ex- 
tremely difficult to construct such a design. These factors particularly affect 
the ability to collect data in a consistent fashion. There are a number of alter- 
natives that are appropriate and can be applied to adult literacy programs. 
Steele ( 1977) has outlined over fifty approaches that can be used in evaluating 
programs that serve disadvantaged adults. One aspect of the measurement 
problem is an informational one, in that program administrators do not have 
the data that they need to make an informed decision about an appropriate 
measurement design. 

Staff Issnei 

Belated to the issues of what and how to measure is the question of who 
is to direct the measurement process. Most adult literacy program administra- 
tors are not trained in evaluation techniques nor do they have access to staff 
who can perform that function The volunteer programs are particularly im- 
peded because of the fluidity of the staffing pattern and the implication that 
this has for the collection of data. Thus a key problem is access to expertise in 
measurement and design. Another issue concerns the staff's perceptions 
about the relationship of evaluation activities to ongoing program functioning. 
Often staff feel that collecting data, either by questionnaires or by testing in- 
struments, can be disruptive to the instructional process. Again, one aspect of 
this problem may be informational when staff have not been included in dis- 
cussions about the type and format of data collection. In addition, the staff 
may not understand the relevance of the information that is being requested 
to what they are doing in facilitating learr ing. A key point here is one of incen- 
tives: if the staff members are critical in aisessing the effectiveness of the pro- 
gram then they need to understand whit the benefits are of participating in 
this activity. 

Relevance 

The focal point of the evaluation activity is usually the adult attending 
the literacy program Sometimes the adults resist participating in assessment 
activities either because these activities are associated with negative experi- 
ences that they have had in prior educational programs, or because they do 
not understand how the evaluation activities relate to the learning process. 
The issue of relevance is particularly important when adults are asked to re- 
spond to questionnaires after they have completed the program. Their willing- 
ness to participate in evaluation activities is often affected by their under- 
standing of the rationale and utility of the process. 

Resource Issues 

A major objection to conducting an evaluation is usually resource-re- 
lated. The human resource problem has already been noted. The issue of the 
availability of adequate financial resources to undertake a proper evaluation is 
one that is not readily solved. The problem should be analyzed so that the var- 
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ious components that are needed can be identified and alternatives devised to 

required that evaluations be conducted. Even with this requirement, program 
^cerosoftenhavetakenpri^ 

duced in scope from the original proposaL The resource issue should be de- 
Uneated so that appropriate solutions can be identified to meet the needs. 

An understanding and recognition of these barriers is critical tf evalua- 
tion is to become an ongoing part of adult Uteracy program* fm^fa" 
rieis have recognizable solutions, others require innovative ticking about the 
™ to wSey can be overcome. In the final section of this paper recom- 

is useful to examine some of the Uteracy program evaluations that have been 
undertaken in recent years in order to understand how some have dealt with 
the issues that have beeti discussed here. 

Recent Evaluation Efforts 

It is difficult to ascertain the magnitude of the effort that has been made 
in evaluating adult Uteracy programs. Little is pubUshed in the educational Ut- 
erature and only one large-scale evaluation has been commissioned in recent 
years by the federal government. An area where significant progress has been 
made is with regard to the U.S. Department of Education's Joint Dissemina- 
tion Review Panel (JDRP) process. As of this time seven adult literacy pro- 
grams have been validated by the JDRP. A number of other adult Uteracy pro- 
grams are in the process of preparing for the JDRP or are beginning to address 
the effectiveness issue by structuring formal evaluations. Pressure from hind- 
ers and a desire to understand the effects of program implerr„ntation have 
prompted the evaluation activity. The various evaluation efforts that nave 
been undertaken will now be discussed. In 1980 Development Associates 
examined the state-administered Adult Basic Education (ABE) programs, in 
which they investigated the activities undertaken by the ABE programs^ in- 
cluding content areas covered in the curriculum, methods and location of in- 
struction, and learner outcomes. The data generated from this study provided 
an informed overview of what has been accomplished by the ABE programs. It 
has also raised a number of issue? that deserve further attention and examina- 
tion Whereas the national study outlined trends among the ABE programs, 
what is needed is a clearer understanding of the factors that contribute to the 
effectiveness of specific programs. The issue might best be explored using a 
case study approach with a sample of selected ABE sites. 

The adult literacy programs that have been vaUdated by the Joint Dis- 
semination Review Panel represent notable attempts at program evaluation. 
While each of the program's evaluation designs differed in structure, certain 
commonalities exist among them. Learner gains were the focal point of the 
APL Project (1979). Project CLASS (1980). FLIT: Functional Literacy (1974). 
Jefferson County Adult Reading Program (1982). and Project F.I.S.T. (1983). 
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while the CAS AS Project U984) and the New York State External High School 
Diploma Program (Alamprese 1979) chose the assessment system as their 
focus. CASAS presented evidence concerning the effectiveness of implement- 
ing a statewide competency-based adult assessment program and the New 
York State External Diploma Program supported its claim of being a reliable 
and valid testing system. Data concerning a program's ability tp retain its stu- 
dents was supplied by CASAS. the Jefferson County Reading Program and the 
External Diploma Program. 

While each of the seven programs had data collection activity built into 
their normal program process, each had to either present the evidence in new 
configurations or collect additional information about program effectiveness 
in order to complete the JDRP submission. This supports a point raised earlier 
in this paper concerning the difficulty of working with an ex post facto design. 
The issue is not whether it is necessary to undertake an evaluation that meets 
the criteria established by the JDRP. but rather that it saves time and resources 
in the long run if one is able to plan a formative evaluation at the outset of a 
program. A number of adult literacy programs that utilize volunteers as their 
primary service deliverers have been dealing with the issue of how to measure 
program effectiveness. For example. Literacy Volunteers of New York City is 
in the process of conducting an evaluation that was designed by Matrices, Inc. 
This evaluation plan takes into consideration the issues of time, staff, and re 
sources and examines the program's context and process in relationship to 
outcomes. Several forms of data collection have been proposed that do not in- 
trude on service delivery and that can be intergrated into the program tunc- 
tioning. Other volunteer programs are focusing on the collection of demo 
graphic data, with the hope of being able to systematically collect information 
on learner outcomes at some point. 2 While demographic information does 
provide a basis for planning recruitment activities and the utilization of staff 
resources and can be incorporated in funding applications, it is not as persua 
sive as data that speak more directly to effectiveness issues. 

The attempts being made by adult literacy programs to include some 
form of evaluation activity in their implementation process are encouraging. 
Most program administrators are confronted with the problem of juggling 
time, resources, and staff in ways that best meet their needs and those of their 
clients. If the trend toward measuring program effectiveness is to continue 
and grow, some assistance in this area will need to be given to adult literacy 
programs. Recommendations regarding the types of assistance that arc 
required to promote the utilization of evaluation are discussed in the 
next section. 

Recommendations 

The paper thus far has examined the evaluation dilemma in adult literacy 
programs in terms of barriers that impede the evaluation process and has ie 
viewed a number of adult literacy programs which have successfully mea- 
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sured and documented their results. The following recommendations con- 
cerning policies and practices that can be implemented to facilitate the evalua- 
tion of adult literacy programs are proposed for consideration. 

1 . Information about evaluation approaches for assessing the effectiveness of 
adult literacy programs should be collected, analyzed, and prepared for dis- 
semination 

A key factor inhibiting the assessment of adult literacy programs is the lack of 
adequate information about appropriate evaluation methods. Alternative ap- 
proaches for agypffldng effectiveness need to be identified and compiled in a 
form mat can be understood and assessed by literacy program administrators. 
That is. strategies already documented such as those cited in Steele (1977). 
and as well those that have been identified in the National Adult Literacy 
Project's survey of effective programs and in related searches, should be as- 
similated and categorized. Having a compendium of evaluation approaches 
that are alternatives to the traditional experimental design would be an initial 
step in facilitating the evaluation process. This activity would most likely re- 
quire discrete funding, such as through a foundation or with Section 310 
monies (Section 310 of the Adult Education Act). As an interim step. Journals 
dealing with adult literacy and adult learning questions could dedicate one 
issue to exploring alternative evaluation strategies. 

2. There should be an increase in the mechanisms available for enhancing the 
skills and abilities of adult literacy staff in measuring, documenting, and 
utilizing program results. 

Training and technical assistance must be made accessible to literacy program 
staff who would like to implement evaluations. This could be in the form of di- 
rect assistance in conducting the evaluation (e.g., how tc choose an evaluation 
approach), or in indirect service, such as brokering information about evalua- 
tion processes. There are several alternatives for providing this technical as- 
sistance. An independent organization with a focus on adult learning could be 
created to deliver direct services and broker information to those engaged in 
literacy activities. On another leveL federal evaluation staff could provide in- 
formation on adult literacy programs that incorporated evaluation approaches 
which can be transferred. State education department staff could be trained to 
provide technical assistance to literacy programs in their state. Evaluation per 
sonnel at the local school district level could be identified and encouraged to 
work with literacy program ctaff in planning evaluations. Finally, evaluation 
workshops could be scheduled at the American Association of Adult and Con- 
tinuing Education (AAACE) and Commission on Adult Basic Education 
(CO ABE) annual meetings to train those interested in assessing program out- 
comes. This recommendation is comprehensive and would require both new 
? resources and a reallocation of existing resources. 

3. Organizations funding adult literacy programs should encourage, if not re- 
quire, documentation of program results and should provide adequate re- 

© sources for this activity. 
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If the quality of teaching and learning in adult literacy programs is to be im- 
proved it is critical to have an understanding of those instructional processes 
and materials that are effective in obtaining results. Literacy programs should 
be encouraged to conduct both formative and summative evaluations. The in- 
formation obtained in the formative evaluation can be used to clarify program 
and client goals and to redirect program activities so that these goals can be 
met A summative evaluation would provide the necessary information to as- 
certain both intended and unintended program results. 
4. Coordinating agencies for the volunteer literacy programs should encour- 
age their affiliates to collect data on program effectiveness and should • 
vide leadership in facilitating the process. 
The volunteer programs that deliver literacy services can be singled out here 
because of the special problems involved in evaluating these programs. 
Leadership is needed to provide assistance and direction to volunteer pro- 
grams so tkit they can overcome the staffing and time problems associated 
with the collection of data. Documented program results in this area would be 
helpful so that effective models could be identified and transferred to those in- 
terested in establishing volunteer programs. 

5 incentives for conducting program evaluations should be clarified and rein- 
forced. 

This recommendation is more general than the others and speaks to an under- 
lying problem that inhibits progress in this area. One reason for the reluctance 
to conduct an evaluation is that the benefits for doing so are not always so 
clear. Literacy program staff should be an integral part of the planning phase 
of the evaluation. They should be given ample information regarding what is 
expected of them and their students and how Ueir program will benefit from 
the evaluation process. Likewise, program administrators have to be better in- 
formed about the ways in which the evaluation data might be analyzed and re- 
ported, and how the program results can be effectively utilized for funding 
and legislative purposes. 

The issues and problems raised in this paper and the recommendations 
that have been proposed are meant to highlight the importance of evaluation 
in adult literacy programs. If the quality and scope of the literacy effort is to in- 
crease, then attention must be focused on improving the evaluation activities 
undertaken. 



Notes 

• l. The discussion presented in this section is based on data the author has collected 
in the course of her work consulting with adult education program administrators 
in evaluation design. 

2. Information concerning the data collection activities of these programs was assem- 
bled by the author as part of her work in another project. 
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